Still Alerted Outside of On Call Schedule

#1

Had an alert on a server from our SolarWinds integration last night. SW sent the alert, PagerDuty accepted the alert, and sent it to the proper Escalation Policy. That policy has a stage 1 of alerting the On Call schedule. And that On Call schedule is only set to run from 3am to 11:30pm - but the alert fired off at 1:18 and 1:19 am, outside of the On Call schedule. It also skipped step 1 of just alerting the On Call (me) and it stepped forward to Step 2 which is to alert 3 people because the first one didn’t respond in 30 minutes… anyone know why the alert would still fire if the On Call schedule isn’t set to start until 3am?

Thanks!

#2

Are you using Support Hours on the PagerDuty Service or just relying on the Escalation Policy and Schedules on each layer?

As described here https://support.pagerduty.com/docs/escalation-policies#escalation-targets-and-limitations, “If nobody is on-call in the first level, we will assign the incident to the next escalation level with an on-call responder.”

May also want to use a rule to set things to Low Urgency during off-hours to prevent escalation up the Escalation Policy.

Doug

#3

Hi Michael,

It does sound like there was a user on-call on the escalation policy which caused the incident to trigger despite level 1 being off-call. The incident would escalate to the next available level if there are gaps in preceding levels.

If you’d like us to take a deeper look into the specific incident, we would be happy to take a look and advise further.

We wouldn’t ask you for the incident here but if you could reach out to Support over email with the incident and perhaps a reference to this Community Post, that would be great.